Multiple sound source encoding in hearing prostheses

ABSTRACT

Presented herein are techniques for enhancing a hearing prosthesis recipient&#39;s perception of multiple frequencies present in received sound signals. The hearing prosthesis is configured to extract a plurality of frequencies from the received sound signals and to use the plurality of frequencies to modulate the amplitudes of different stimulation pulse sequences that are to be delivered to the recipient via different stimulation channels. The hearing prosthesis may also adapt a stimulation resolution of the stimulation pulse sequences when delivering the modulated stimulation pulses sequences to the recipient.

BACKGROUND Field of the Invention

The present invention relates generally to electrically-stimulatinghearing prostheses.

Related Art

Hearing loss, which may be due to many different causes, is generally oftwo types, conductive and/or sensorineural. Conductive hearing lossoccurs when the normal mechanical pathways of the outer and/or middleear are impeded, for example, by damage to the ossicular chain or earcanal. Sensorineural hearing loss occurs when there is damage to theinner ear, or to the nerve pathways from the inner ear to the brain.

Individuals who suffer from conductive hearing loss typically have someform of residual hearing because the hair cells in the cochlea areundamaged. As such, individuals suffering from conductive hearing losstypically receive a hearing prosthesis that generates motion of thecochlea fluid. Such auditory prostheses include, for example, acoustichearing aids, bone conduction devices, and direct acoustic stimulators.

In many people who are profoundly deaf, however, the reason for theirdeafness is sensorineural hearing loss. Those suffering from some formsof sensorineural hearing loss are unable to derive suitable benefit fromauditory prostheses that generate mechanical motion of the cochleafluid. Such individuals can benefit from implantable auditory prosthesesthat stimulate nerve cells of the recipient's auditory system in otherways (e.g., electrical, optical and the like). Cochlear implants areoften proposed when the sensorineural hearing loss is due to the absenceor destruction of the cochlea hair cells, which transduce acousticsignals into nerve impulses. An auditory brainstem stimulator is anothertype of stimulating hearing prosthesis that might also be proposed whena recipient experiences sensorineural hearing loss due to damage to theauditory nerve.

Certain individuals suffer from only partial sensorineural hearing lossand, as such, retain at least some residual hearing. These individualsmay be candidates for electro-acoustic hearing prostheses.

SUMMARY

In one aspect, a method is provided. The method comprises: receivingsound signals at a hearing prosthesis; extracting a plurality offrequencies from the sound signals; filtering the sound signals togenerate channelized sound signals; determining a plurality ofstimulation pulse sequences, wherein each stimulation pulse sequencescorresponds to one of the channelized sound signals; amplitudemodulating each of the plurality of stimulation pulse sequences with oneof the plurality of frequencies extracted from the sound signals,wherein at least two of the plurality of stimulation pulse sequences areamplitude modulated with different ones of the plurality of frequenciesextracted from the sound signals; and delivering each of the pluralityof stimulation pulse sequences to the recipient via one or morestimulation channels of the hearing prosthesis.

In another aspect, a method is provided. The method comprises: receivingsound signals at a hearing prosthesis; extracting at least one frequencyfrom the sound signals; filtering the sound signals to generatechannelized sound signals; determining a plurality of stimulation pulsesequences, wherein each of the plurality of stimulation pulse sequencescorresponds to one of the channelized sound signals; determining aperiodic probability for each of a plurality of the channelized soundsignals, wherein a periodic probability indicates a degree ofassociation between a channelized sound signal and the at least onefrequency extracted from the sound signals; and amplitude modulating atleast one of the plurality of stimulation pulse sequences based on aperiodic probability associating a channelized sound signalcorresponding to the at least one stimulation pulse sequence and the atleast one frequency extracted from the sound signals.

In another aspect, a hearing prosthesis is provided. The hearingprosthesis comprises: one or more sound input elements configured toreceive sound signals; a memory; a stimulator unit; and at least oneprocessor configured to: estimate at least first and second differentfrequencies present within the received sound signals, determine atleast first and second stimulation pulse sequences based on the soundsignals, amplitude modulate the first stimulation pulse sequence basedon the first frequency, and amplitude modulate the second stimulationpulse sequence based on the second frequency, wherein the stimulatorunit is configured to deliver the first and second stimulation pulsesequences to a recipient of the hearing prosthesis.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are described herein in conjunctionwith the accompanying drawings, in which:

FIG. 1A is a schematic diagram illustrating a cochlear implant, inaccordance with certain embodiments presented herein;

FIG. 1B is a block diagram of the cochlear implant of FIG. 1A;

FIG. 2 is a block diagram of a totally implantable cochlear implant, inaccordance with certain embodiments presented herein;

FIG. 3 is a functional block diagram of a sound processing block, inaccordance with certain embodiments presented herein;

FIG. 4 is a flowchart of a method in accordance with embodimentspresented herein

FIG. 5 is a functional block diagram illustrating channel modulationusing a single sound source, in accordance with certain embodimentspresented herein;

FIG. 6 is a functional block diagram illustrating channel modulationusing multiple sound sources, in accordance with certain embodiments;

FIGS. 7A, 7B, 7C, 7D, and 7E are schematic diagrams illustrating theadaption of spatial resolution of electrical stimulation signals basedon periodic probabilities, in accordance with certain embodimentspresented herein;

FIG. 8 is an electrodogram illustrating three sequences of fixed ratestimulation pulses modulated based on channel envelopes;

FIG. 9 is an electrodogram illustrating thee sequences of fixed ratestimulation pulses modulated using a single extracted fundamentalfrequency;

FIG. 10 is an electrodogram illustrating thee sequences of fixed ratestimulation pulses modulated using three different extracted fundamentalfrequencies;

FIG. 11 illustrates an example scenario in which a hearing prosthesisreceives sound signals that include multiple sound sources;

FIG. 12 illustrates another example scenario in which a hearingprosthesis receives sound signals that include multiple sound sources;

FIG. 13 is a block diagram of a sound processing unit in accordance withembodiments presented herein;

FIG. 14 is a flowchart of a method in accordance with embodimentspresented herein; and

FIG. 15 is flowchart of another method in accordance with embodimentspresented herein.

DETAILED DESCRIPTION

Embodiments of the present invention are generally directed totechniques for enhancing a hearing/auditory prosthesis recipient'sperception of multiple frequencies (e.g., multiple fundamentalfrequencies) present in received sound signals. The hearing prosthesisis configured to extract a plurality of frequencies from the receivedsound signals and to use the plurality of frequencies to modulate theamplitudes of different stimulation pulse sequences that are to bedelivered to the recipient via different stimulation channels. Thehearing prosthesis may also adapt a stimulation resolution of thestimulation pulse sequences when delivering the modulated stimulationpulses sequences to the recipient.

There are a number of different types of electrically-stimulatingauditory/hearing prostheses in which embodiments of the presentinvention may be implemented. However, merely for ease of illustration,the techniques presented herein are primarily described with referenceto one type of electrically-stimulating hearing prosthesis, namely acochlear implant. However, it is to be appreciated that the techniquespresented herein may be used in other electrically-stimulating auditoryprostheses, such as auditory brainstem stimulators, electro-acoustichearing prostheses, bimodal hearing prostheses, etc.

FIG. 1A is a schematic diagram of an exemplary cochlear implant 100configured to implement aspects of the techniques presented herein,while FIG. 1B is a block diagram of the cochlear implant 100. For easeof illustration, FIGS. 1A and 1B will be described together.

The cochlear implant 100 comprises an external component 102 and aninternal/implantable component 104. The external component 102 isdirectly or indirectly attached to the body of the recipient andtypically comprises an external coil 106 and, generally, a magnet (notshown in FIG. 1) fixed relative to the external coil 106. The externalcomponent 102 also comprises one or more input elements/devices 113 forreceiving input sound signals at a sound processing unit 112. In thisexample, the one or more input devices 113 include microphones 108positioned by auricle 110 of the recipient configured to capture/receiveinput sound signals, one or more auxiliary input devices 109 (e.g.,audio ports, such as a Direct Audio Input (DAI), data ports, such as aUniversal Serial Bus (USB) port, cable port, etc.), and a wirelesstransmitter/receiver (transceiver) 111, each located in, on, or near thesound processing unit 112. Although not shown in FIG. 1A or 1B, theinput devices 113 could also include, for example, telecoils or othertypes of inputs.

The sound processing unit 112 also includes, for example, at least onebattery 107, a radio-frequency (RF) transceiver 121, and amulti-frequency sound processor 125. The multi-frequency sound processor125 may be formed by one or more processors (e.g., one or more DigitalSignal Processors (DSPs), one or more uC cores, etc.), memories,firmware, software, etc. arranged to perform operations describedherein. That is, the multi-frequency sound processor 125 may beimplemented as firmware elements, partially or fully implemented withdigital logic gates in one or more application-specific integratedcircuits (ASICs), by processors executing software (instructions) storedin memory, etc.

Returning to the specific example of FIGS. 1A and 1B, the implantablecomponent 104 comprises an implant body (main module) 114, a lead region116, and an intra-cochlear stimulating assembly 118, all configured tobe implanted under the skin/tissue (tissue) 105 of the recipient. Theimplant body 114 generally comprises a hermetically-sealed housing 115in which RF interface circuitry 124 and a stimulator unit 120 aredisposed. The implant body 114 also includes an internal/implantablecoil 122 that is generally external to the housing 115, but which isconnected to the RF interface circuitry 124 via a hermetic feedthrough(not shown in FIG. 1B).

As noted, stimulating assembly 118 is configured to be at leastpartially implanted in the recipient's cochlea 137. Stimulating assembly118 includes a plurality of longitudinally spaced intra-cochlearelectrical stimulating contacts (electrodes) 126 that collectively forma contact or electrode array 128 for delivery of electrical stimulation(current) to the recipient's cochlea. Stimulating assembly 118 extendsthrough an opening in the recipient's cochlea (e.g., cochleostomy, theround window, etc.) and has a proximal end connected to stimulator unit120 via lead region 116 and a hermetic feedthrough (not shown in FIG.1B). Lead region 116 includes a plurality of conductors (wires) thatelectrically couple the electrodes 126 to the stimulator unit 120.

As noted, the cochlear implant 100 includes the external coil 106 andthe implantable coil 122. The coils 106 and 122 are typically wireantenna coils each comprised of multiple turns of electrically insulatedsingle-strand or multi-strand platinum or gold wire. Generally, a magnetis fixed relative to each of the external coil 106 and the implantablecoil 122. The magnets fixed relative to the external coil 106 and theimplantable coil 122 facilitate the operational alignment of theexternal coil with the implantable coil. This operational alignment ofthe coils 106 and 122 enables the external component 102 to transmitdata, as well as possibly power, to the implantable component 104 via aclosely-coupled wireless link formed between the external coil 106 withthe implantable coil 122. In certain examples, the closely-coupledwireless link is a radio frequency (RF) link. However, various othertypes of energy transfer, such as infrared (IR), electromagnetic,capacitive and inductive transfer, may be used to transfer the powerand/or data from an external component to an implantable component and,as such, FIG. 1B illustrates only one example arrangement.

As noted above, sound processing unit 112 includes the multi-frequencysound processor 125, which may be implemented in hardware, software,and/or a combination thereof. In general, the multi-frequency soundprocessor 125 is configured to convert input audio signals intostimulation control signals 130 for use in stimulating a first ear of arecipient (i.e., the processing block 125 is configured to perform soundprocessing on input audio signals received at the sound processing unit112). Stated differently, the multi-frequency sound processor 125 (e.g.,one or more processing elements implementing firmware, software, etc.)is configured to convert the captured input audio signals intostimulation control signals 130 that represent electrical stimulationfor delivery to the recipient. The input audio signals that areprocessed and converted into stimulation control signals may be audiosignals received via the sound input devices 108, signals received viathe auxiliary input devices 109, and/or signals received via thewireless transceiver 111.

In accordance with certain embodiments presented herein, to generate thestimulation control signals 130, the multi-frequency sound processor 125is configured to identify and track multiple sound sources (track soundcomponents associated with received sound signals), as well as toextract at least one frequency (e.g., the fundamental frequency) of eachof the sound sources. The multi-frequency sound processor 125 is alsoconfigured to generate the stimulation control signals 130 such that themultiple frequencies extracted from the sound signals are encoded in thefinal stimulation pulse sequences that are delivered to the recipient.The multi-frequency sound processor 125 may also enable the differentfrequencies to be individually controlled.

In the embodiment of FIG. 1B, the stimulation control signals 130 areprovided to the RF transceiver 121, which transcutaneously transfers thestimulation control signals 130 (e.g., in an encoded manner) to theimplantable component 104 via external coil 106 and implantable coil122. That is, the stimulation control signals 130 are received at the RFinterface circuitry 124 via implantable coil 122 and provided to thestimulator unit 120. The stimulator unit 120 is configured to utilizethe stimulation control signals 130 to generate electrical stimulationsignals (e.g., sequences of stimulation (current) pulses) for deliveryto the recipient's cochlea via one or more stimulation channels eachformed by one or more stimulating contacts 126. In this way, cochlearimplant 100 electrically stimulates the recipient's auditory nervecells, bypassing absent or defective hair cells that normally transduceacoustic vibrations into neural activity, in a manner that causes therecipient to perceive one or more components of the input audio signals.

FIGS. 1A and 1B illustrate an arrangement in which the cochlear implant100 includes an external component. However, it is to be appreciatedthat embodiments of the present invention may be implemented in cochlearimplants or auditory prostheses having alternative arrangements. Forexample, FIG. 2 is a functional block diagram of an exemplary totallyimplantable cochlear implant 200 configured to implement embodiments ofthe present invention. Since the cochlear implant 200 is totallyimplantable, all components of cochlear implant 200 are configured to beimplanted under skin/tissue 205 of a recipient. Because all componentsare implantable, cochlear implant 200 operates, for at least a finiteperiod of time, without the need of an external device. An externaldevice 202 can be used to, for example, charge an internal power source(battery) 207. External device 202 may be, for example, a dedicatedcharger or a conventional cochlear implant sound processor. 100381Cochlear implant 200 includes an implant body (main implantablecomponent) 214, one or more input elements 213 for capturing/receivinginput audio signals (e.g., one or more implantable microphones 208 and awireless transceiver 211), an implantable coil 222, and an elongateintra-cochlear stimulating assembly 118 as described above withreference to FIGS. 1A and 1B. The microphone 208 and/or the implantablecoil 222 may be positioned in, or electrically connected to, the implantbody 214. The implant body 214 further comprises the battery 207, RFinterface circuitry 224, a multi-frequency sound processor 225, and astimulator unit 220 (which is similar to stimulator unit 120 of FIGS. 1Aand 1B). The multi-frequency sound processor 225 may be similar tomulti-frequency sound processor 125 of FIGS. 1A and 1B.

In the embodiment of FIG. 2, the one or more implantable microphones 208are configured to receive input audio signals. The multi-frequency soundprocessor 225 is configured to convert received signals into stimulationcontrol signals 230 for use in stimulating a first ear of a recipient.Stated differently, the multi-frequency sound processor 225 isconfigured to convert the input audio signals into stimulation controlsignals 230 that represent electrical stimulation for delivery to therecipient. Similar to the multi-frequency sound processor 125 of FIGS.1A and 1B, to generate the stimulation control signals 230, themulti-frequency sound processor 225 is configured to identify and trackmultiple sound sources (sound components associated with sound sources),as well as to extract at least one frequency (e.g., the fundamentalfrequency) associated with each of the sound sources. Themulti-frequency sound processor 225 is also configured to generate thestimulation control signals 230 such that the multiple frequencies areencoded in the final stimulation pulse sequences that are delivered tothe recipient. The multi-frequency sound processor 225 may also enablethe different frequencies to be individually controlled.

As noted above, FIGS. 1A and 1B illustrate an embodiment in which theexternal component 102 includes the multi-frequency sound processor 125.As such, in the illustrative arrangement of FIGS. 1A and 1B, thestimulation control signals 130 are provided to the implanted stimulatorunit 120 via the RF link between the external coil 106 and the internalcoil 122. However, in the embodiment of FIG. 2 the multi-frequency soundprocessor 225 is implanted in the recipient. As such, in the embodimentof FIG. 2, the stimulation control signals 230 do not traverse the RFlink, but instead are provided directly to the stimulator unit 220. Thestimulator unit 220 is configured to utilize the stimulation controlsignals 230 to generate electrical stimulation signals (e.g., sequencesof stimulation (current) pulses) that are delivered to the recipient'scochlea via one or more stimulation channels.

A recipient's cochlea is tonotopically mapped, that is, partitioned intoregions each responsive to sound signals in a particular frequencyrange. In general, the basal region of the cochlea is responsive tohigher frequency sounds, while the more apical regions of the cochleaare responsive to lower frequency sounds. The tonopotic nature of thecochlea is leveraged in cochlear implants such that specific acousticfrequencies are allocated to the electrode(s) of the stimulatingassembly that are positioned close to the corresponding tonotopic regionof the cochlea (i.e., the region of the cochlea that would naturally bestimulated in acoustic hearing by the acoustic frequency). That is, in acochlear implant, specific frequency bands are each mapped to a set ofone or more electrodes that are used to stimulate a selected (target)population of cochlea nerve cells. The frequency bands, and associatedelectrode(s), form a “stimulation channel” that delivers stimulationsignals to the recipient. During operation, a cochlear implant soundprocessor encodes or maps different frequency portions of the receivedsound signals sound signals to the electrodes that should be used todeliver stimulation signals representing the different frequencyportions.

Certain conventional sound encoding (coding) strategies forelectrically-stimulating auditory prostheses are effective in enabling arecipient to correctly perceive the fundamental frequency (F0)associated with a single source of sound (sound source). However, manyreal-world environments include multiple sound sources with differentfundamental frequencies, or may comprise a single sound source thatincludes multiple harmonics or concurrent pitches. Conventional soundcoding strategies generally lack the ability to appropriately captureand deliver multiple fundamental frequencies (F0s) from these differentsound sources and/or multiple harmonics included in a single soundsource. Consequently, recipients may not have access to the auditorycues necessary for accurate perception of multiple frequencies inreceived sound signals, such as, musical harmonies, musical compositionswith multiple instruments, speech signals from multiple talkers, or inother situations with multiple sound sources, multiple harmonics, and/orconcurrent pitches.

Accordingly, presented herein are techniques that improve a recipient'sperception of multiple frequencies (e.g., multiple F0s, multipleharmonics, etc.) included within sound signals received at a hearingprosthesis (i.e., multiple frequencies simultaneously received at theprosthesis). As described further below, the techniques presented hereinidentify and track sound components associated with different soundsources that are included within received sound signals, extractfrequencies of each sound source, encode the multiple frequencies withinstimulation signals delivered to the recipient, and allow the differentfrequencies to be individually controlled. As noted, these techniquesmay be implemented by a multi-frequency sound processor, such asmulti-frequency sound processors 125 and 225, of anelectrically-stimulating hearing prosthesis.

FIG. 3 is a functional block diagram illustrating an examplemulti-frequency sound processor configured to implement the techniquespresented herein. For ease of description, the multi-frequency soundprocessor of FIG. 3 is referred to as multi-frequency sound processor325.

In the embodiment of FIG. 3, the multi-frequency sound processor 325comprises: a filterbank module 334, an envelope extraction module 336, achannel selection module 338, and a stimulation pulse determinationmodule 340. Modules 334, 336, 338, and 340 are sometimes collectivelyreferred to herein as a “sound processing path” 358 for themulti-frequency sound processor 325. However, in addition to the soundprocessing path 358 (i.e., modules 336, 338, and 340), themulti-frequency sound processor 325 also comprises a fundamentalfrequency module 342, an environmental classification module 344, aperiodic probability estimator module 346, a channel modulation module348, a channel focusing control module 350, an automatic control module352, and a user control module 354.

It is to be appreciated that the functional modules shown in FIG. 3generally represent functions/operations that can be performed inaccordance with embodiments presented herein, and do not necessarilyimply any specific structure for the multi-frequency sound processor325. For example, the modules shown in FIG. 3 may be implemented asfirmware elements, partially or fully implemented with digital logicgates in one or more application-specific integrated circuits (ASICs),by processors executing software (instructions) stored in memory, etc.As described further below, it is also to be appreciated that thespecific combination of functional modules shown in FIG. 3 areillustrative and that multi-frequency sound processors in accordancewith certain embodiments may include only a subset of the modules shownin FIG. 3 and/or other functional modules that, for ease ofillustration, have been omitted from FIG. 3.

In the example of FIG. 3, sound signals are received/captured by one ormore input devices (e.g., microphones, audio input port, etc.) andprovided as input sound signals 331 to a filterbank module (filterbank)334. The input sound signals 331 may also be provided to the fundamentalfrequency module 342 and the environmental classification module 344.Although not shown in FIG. 3, in certain embodiments a pre-filterbankprocessing module may be present and configured to, as needed, combinesignals from different sound input elements to generate the input soundsignals 331 and to, again as needed, prepare those signals forsubsequent processing by the filterbank module 334, etc.

In operation, the filterbank module 334 generates a suitable set ofbandwidth limited channels, or frequency bins, that each includes aspectral component of the received sound signals. That is, thefilterbank module 334 may comprise a plurality of band-pass filters thatseparate the input sound signals 331 into multiple bands/channels, eachone carrying a single frequency sub-band of the original signal (i.e.,frequency components of the received sounds signal).

The channels created by the filterbank module 334 are sometimes referredto herein as sound processing channels, and the sound signal componentswithin each of the sound processing channels are sometimes referred toherein as band-pass filtered signals or channelized signals. Theband-pass filtered or channelized signals created by the filterbankmodule 334 are processed (e.g., modified/adjusted) as they pass throughthe sound processing path 358. As such, the band-pass filtered orchannelized signals are referred to differently at different stages ofthe sound processing path 358. However, it will be appreciated thatreference herein to a band-pass filtered signal or a channelized signalmay refer to the spectral component of the received sound signals at anypoint within the sound processing path 358 (e.g., pre-processed,processed, selected, etc.).

At the output of the filterbank module 334, the channelized signals areinitially referred to herein as pre-processed signals 335. The number‘m’ of channels and pre-processed signals 335 generated by thefilterbank module 334 may depend on a number of different factorsincluding, but not limited to, implant design, number of activeelectrodes, coding strategy, and/or recipient preference(s). In certainarrangements, twenty-two (22) channelized signals are created and thesound processing path 358 is said to include 22 channels.

The pre-processed signals 335 are provided to the envelope extractionmodule 336, which determines/extracts the amplitude envelopes 337 of theprocessed signals 335 within each of the channels. These envelopes 337are provided to the channel selection module 338, as well as to thechannel modulation module 348. The channel selection module 338 isconfigured to perform a channel selection process to select, accordingto one or more selection rules, which of the ‘m’ channels should be usein hearing compensation. The signals selected at channel selectionmodule 338 are represented in FIG. 3 by arrow 339 and are referred toherein as selected channelized signals or, more simply, selectedsignals.

In the embodiment of FIG. 3, the channel selection module 338 selects asubset ‘n’ of the ‘m’ envelopes 337 for use in generation of electricalstimulation for delivery to a recipient (i.e., the sound processingchannels are reduced from ‘m’ channels to ‘n’ channels). In one specificexample, the ‘n’ largest amplitude channels (maxima) from the ‘m’available combined channel signals/masker signals is made, with ‘m’ and‘n’ being programmable during initial fitting, and/or operation of theprosthesis. It is to be appreciated that different channel selectionmethods could be used, and are not limited to maxima selection.

It is also to be appreciated that, in certain embodiments, the channelselection module 338 may be omitted. For example, certain arrangementsmay use a continuous interleaved sampling (CIS), CIS-based, or anothernon-channel selection sound coding strategy.

The stimulation pulse determination module 340 is configured to map theamplitudes of the selected signals 339 (or the envelopes 337 inembodiments that do not include channel selection) into a set of outputsignals 330 (e.g., stimulation commands) that represent the attributesof the electrical stimulation signals that are to be delivered to therecipient so as to evoke perception of at least a portion of thereceived sound signals. This channel mapping may include, for example,threshold and comfort level mapping, dynamic range adjustments (e.g.,compression), volume adjustments, etc., and may encompass selection ofvarious sequential and/or simultaneous stimulation strategies. Furtherdetails regarding the operation of the stimulation pulse determinationmodule 340 are provided below.

Although not shown in FIG. 3, the sound processing path 358 may includea processing module that is configured to perform a number of soundprocessing operations on the channelized signals. These sound processingoperations include, for example, channelized gain adjustments forhearing loss compensation (e.g., gain adjustments to one or morediscrete frequency ranges of the sound signals), noise reductionoperations, speech enhancement operations, etc., in one or more of thechannels. In certain implementations, these operations may be performedin the stimulation pulse determination module 340.

As noted above, in addition to modules 336, 338, and 340, themulti-frequency sound processor 325 also comprises the fundamentalfrequency module 342, the environmental classification module 344, theperiodic probability estimator module 346, the channel modulation module348, the channel focusing control module 350, the automatic controlmodule 352, and the user control module 354. Each of these modules 342,344, 346, 348, 350, 352, and 354, may perform supplemental operationsthat, in accordance with the techniques presented herein, control oraffect the operations performed in the sound processing path 358 togenerate the stimulation control signals 330. For ease of description,the operation of modules 342, 344, 346, 348, 350, 352, and 354, aredescribed further below with reference to FIGS. 4-12.

More specifically, FIG. 4 is a high-level flowchart of a method 460 inaccordance with embodiments presented herein. Method 460 begins 461where the environmental classification module 344 evaluates/analyzes theinput sound signals and determines the sound class/category/environmentof the sound signals. That is, the environmental classification module344 is configured to use the received sound signals to “classify” theambient sound environment and/or the sound signals into one or moresound categories (i.e., determine the input signal type). The soundclass or enviroment may include, but are not limited to, “Speech,”“Noise,” “Speech+Noise,” “Wind,” “Music,” and “Quiet.” The environmentalclassification module 344 may also estimate the signal-to-noise ratio(SNR) of the sound signals. In one example, the operations of theenvironmental classification module 344 are performed using the inputsound signals 331. The environmental classification module 344 generatessound classification information/data 341. The sound classification data341 represents the sound class of the sound signals and, in certainexamples, the SNR of the sound signals.

In certain examples, the environmental classification module 344operates a gating function for the techniques presented herein. That is,certain sound classes may benefit from pitch and spectral resolutionenhancements (e.g., Music, Speech, speech-in-noise), while others (e.g.,Wind, Noise, Quiet) may not. If the environmental classification module344 determines that the sound class of the input sound signals matches asound class that can benefit from pitch and spectral resolutionenhancements, then the sound classification data 341 can be provided tothe fundamental frequency module 342 to trigger subsequent operations.Stated differently, in certain examples, the operations described belowwith reference to 462-467 may be performed only when the sound signalscorrespond to certain sound classes, while other sound signals will beprocessed according to standard techniques (e.g., advanced combinationencoders (ACE)/continuous interleaved sampling-like strategies withmonopolar stimulation).

Returning to FIG. 4, at 462 the fundamental frequency module 342estimates one or more (e.g., multiple) fundamental frequencies (F0s) ofthe input sound signals. That is, the fundamental frequency module 342is configured to analyze the input sound signals and extract one or morefundamental frequencies present therein. A number of differenttechniques that are based on temporal methods, spectral methods,statistical methods or data-driven approaches can be used for thispurpose.

At 463, the fundamental frequency module 342 is configured to track thesound sources across time (i.e., track, over time, sound componentsassociated with each of the plurality of sound sources) based on theirassociated F0 so that a specific sound source (i.e., sound componentsassociated with a particular sound source) can be delivered to the samechannel (set of electrodes) or the same ear, a specific sound source tobe tracked over time, including variations in F0 attributed to the samesound source. For example, methods based on temporal continuity criteriaor amplitude modulation cues could be used to track multiple sources/F0sacross time.

As used herein, reference to a sound source as being “included in,” orotherwise as being part of, the received sound signals is to beconstrued as reference to received sound components of the sound signalsthat are generated by the sound source. For example, reference to soundsignals that include multiple sound sources refers to sound signals thatinclude sound components each generated by one of the multiple soundsources.

For example, in certain embodiments, spectral filtering of harmonics maybe used to identify and track the fundamental frequencies (F0s) of theinput sound signals. In such examples, the fundamental frequency module342 detects all salient spectral peaks while the spectrum typicallycontains some low-level broadband energy due to noise and spectralleakage. The fundamental frequency module 342 estimates pitchtrajectories over all time frames using a multi-pitch estimator andmatches spectral peaks to harmonics. Fitters are constructed to separatethe spectral peaks or “harmonics” associated with one of the extractedpitches from the mixed spectrum

In other embodiments, temporal autocorrelation functions may be used toidentify and track the fundamental frequencies (F0s) of the input soundsignals. In such examples, the fundamental frequency module 342 dividesthe input sound signals into two channels or ranges (e.g., below andabove 1000 Hertz (Hz)). The fundamental frequency module 342 computes a“generalized” autocorrelation of the low-channel signal and of theenvelope of the high-channel signal, and sums the autocorrelationfunctions. The summary autocorrelation function (SACF) is furtherprocessed to obtain an enhanced SACF (ESACF). The SACF and ESACFrepresentations are used in observing the periodicities of the inputsignal.

In other embodiments, harmonicity and spectral smoothness may be used toidentify and track the fundamental frequencies (F0s) of the input soundsignals. In such examples, the fundamental frequency module 342 operatesiteratively by estimating and removing the most prominent F0 from themixture signal. The term predominant-F0 estimation refers to a crucialstage where the F0 of the most prominent sound is estimated in thepresence of other harmonic and noisy sounds. To achieve this, theharmonic frequency relationships of simultaneous spectral components areused to group them to sound sources. An algorithm is proposed which isable to handle inharmonic sounds. These are sounds for which thefrequencies of the overtone partials (harmonics) are not in exactinteger ratios. In a subsequent stage, the spectrum of the detectedsound is estimated and subtracted from the mixture. This stage utilizesthe spectral smoothness principle, which refers to the expectation thatthe spectral envelopes of real sounds tend to be slowly varying as afunction of frequency. In other words, the amplitude of a harmonicpartial is usually close to the amplitudes of the nearby partials of thesame sound. The estimation and subtraction steps are then repeated forthe residual signal.

It is to be appreciated that the above techniques for identifying andtracking the fundamental frequencies (F0s) of the input sound signalsare merely illustrative and that embodiments presented herein mayalternatively make use of a number of other techniques. Regardless ofthe technique used, the fundamental frequency module 342 provides theidentified and tracked sound sources, represented in FIG. 3 by arrow343, to the periodic probability estimator 346.

Returning to FIG. 4, at 464 the periodic probability estimator 346 isconfigured to estimate the periodic probability of the channels presentin the pre-processed signals 335 generated by the filterbank module 334.That is, the periodic probability estimator 346 is used to estimate theprobability that the signal (e.g., harmonics) present in any givenfrequency channel is related to an estimated F0 frequency (i.e.,contains frequency components, or partials, that are an integer multipleof the estimated F0 frequency, and/or contains periodicity in itsenvelope that is equal to the estimated F0 frequency). Thisfunctionality could be extended so that harmonics in each channel arecompared against all the available F0s in the signal to obtain theperiodic probabilities for all F0s. These probability estimates can befurther used to separate out sources corresponding to the individual F0sthat were identified by the fundamental frequency module 342.

In certain embodiments, the periodic probability estimator 346 employstwo methods, one for low-frequency channels (e.g., 0-2 kHz) and adifferent one for high-frequency channels (e.g., 2 kHz). Forlow-frequency channels, the periodic probability estimator 346calculates the ratio of the power in the harmonic frequency bins to thetotal power within that channel. A Gaussian sieve process is used tofilter the harmonic regions of the bandwidth within each channel. Next,the above ratios are scaled by the total power of the low-frequencychannels (0-2 kHz) to obtain the probability of the channel containingthe harmonic of an estimated F0. For high-frequency channels, thechannel periodic probability may be estimated by determining whether theperiod of the channel envelope signal is equal to (or close to) theperiod of the estimated F0 frequency. This is achieved by high-passfiltering the wide-bandwidth channel envelope signal obtained from thefilterbank module, and maintaining a history of it in a buffer (e.g., ofapproximately 28 ms duration).

It is to be appreciated that the above techniques for estimating theperiodic probability for a given channel are merely illustrative andthat embodiments presented herein may alternatively make use of a numberof other techniques. Regardless of the technique used, the periodicprobability estimator module 346 provides the periodic probability (orprobabilities) determined for each channel to the channel modulationmodule 348. In FIG. 3, the determined periodic probability (orprobabilities), which are represented by arrow 345, are provided to thechannel modulation module 348.

In the case of multiple F0s, the fundamental frequency module 342 willoutput one or more F0s. In such examples, the periodic probabilityestimator 346 can implement the above processes repeatedly for eachestimated F0. That is, for an M-channel cochlear implant, there will beM periodic probabilities per estimated F0. For example, for 2 differentF0s, there will be m periodic probabilities for the first F0 and asecond set of M periodic probabilities for the second F0. In otherwords, each channel will have two probabilities (or K probabilities)corresponding to the two sources (or K sources).

Returning to FIG. 4, at 465 the channel modulation module 348 isconfigured to adapt channel amplitude modulations applied in the soundprocessing path according to the periodic probability determined for thegiven channel. More specifically, as noted, the envelope extractionmodule 336 determines the channel envelope signals 337 (envelopes) fromthe bandpass filtered signals 335, and slowly-varying channel envelopesignals are further extracted from the original set of channelenvelopes. The slowly-varying channel envelope signals are amplitudemodulated by a given frequency, determined by the fundamental frequencymodule 342. In certain examples, the modulated channel envelopes,represented in FIG. 3 by arrow 347, are mixed with the non-modulatedchannel envelopes at a mixing ratio determined by the associatedperiodic probability. An example of such a process is shown in FIG. 5(i.e., a Sound Source “A” is identified and the associated estimatedperiodic probability, for a given channel, is used to control the mixingratio for that channel).

The sequences of stimulation pulses in each of the individual channelscan be amplitude modulated according to a number of different ways and,in certain embodiments, the amplitude modulations may be based onmultiple identified frequencies (e.g., multiple F0s, multiple harmonics,etc.). For example, the sequences of stimulation pulses in each channelmay be amplitude modulated with different F0 scorresponding to separatesound sources or with different frequencies that are harmonics of the F0of a single sound source. The dominant sound source in a given frequencychannel, as determined by the amplitude of the sound source, may be usedto select the F0 for use in modulating the pulses in that channel.Alternatively, channels may be grouped and allocated to different soundsources. For example, FIG. 6 illustrates an example of such a process inwhich the periodic probabilities associated with multiple sources areused to control a mixing ratio for mixing modulated channel envelopeswith non-modulated channel envelopes (i.e., Sound Sources “A,” “B,” “C,”and “D” are identified and the estimated periodic probabilities, for agiven channel, associated with one or more of these sound sources isused to control the mixing ratio for that channel).

In one embodiment in which multiple F0s are identified, the sequences ofstimulation pulses in each channel can be amplitude modulated inaccordance with (based on) the source F0 that results in maximumperiodic probability among all the source F0s for that channel (i.e.,channel modulation based on strength of harmonicity). In otherembodiments in which multiple F0s are identified, the sequences ofstimulation pulses in each channel can be modulated based on the F0 forthe source that is selected by the user (e.g., via user input module 354in FIG. 3) or through an automatic control (e.g., via automatic controlmodule 352), sometimes referred to as channel modulation based on sourceselection. In further embodiments in which multiple F0s are identified,the sequences of stimulation pulses in all of the channels may bemodulated based on the source that results in a greater number of higherperiodic probabilities across all the channels. For example, if twelve(12) out of twenty (20) channels result in higher periodic probabilitiesfor source B compared to that of source A, then the sequences ofstimulation pulses in all 20 channels may be amplitude modulatedaccording to the periodic probabilities and F0 corresponding to sourceB. In another embodiment, all channels can be modulated according to theperiodic probabilities of the source that corresponds to the largestperiodic probability on any channel (i.e., channel modulations based ondominant source/channel).

Returning again to FIG. 4, at 466 the channel focusing control module350 is configured to adapt a stimulus resolution of the sequence ofmodulated stimulation pulses, when delivered via a stimulation channel.That is, certain embodiments presented herein combine the use ofperiodic probability estimation and F0 channel modulation with stimulusresolution adaptions, such as stimulation focusing, to provideadditional frequency perception (pitch) enhancements. For example, itmay be desirable to deliver focused stimulation for periodic signalssuch as vowels and melodic musical instruments to improve the perceptionof simultaneously received formants and harmonics. However, non-periodicsignals, such as unvoiced consonants and percussive musical instruments,may not benefit as much from focused stimulation. Therefore, the degreeof stimulus resolution adaption (e.g., focusing) is reduced fornon-periodic sounds, resulting in more monopolar-like stimulation andpotential power savings. In certain embodiments, the stimulus resolutionadaption is implemented by adjusting the defocusing index, which rangesbetween 0 (i.e., fully focused stimulation) and 1 (i.e., monopolarstimulation).

Improved channel independence via focused stimulation may also providebetter representation of F0 modulations, particularly when different F0sare represented across different channels. Therefore, adaptive focusingmay be further combined with the channel modulations described above toexplicitly represent F0s and/or harmonics estimated from the soundsignals. Channels with high periodic probability will be stimulated witha lower defocusing index and a higher mixing ratio of the modulatedchannel envelope, while channels with low periodic probability willreceive a higher defocusing index and a lower mixing ratio.

In one example, the channel focusing control module 350 may adjustoperations of the stimulation pulse determination module 340 to set thestimulus resolution of the delivered electrical stimulation signals(pulse sequences) based on an associated periodic probability. In oneembodiment, the spatial/spectral attributes of the stimulus resolutionare set by switching between different channel/electrode configurations,such as between monopolar stimulation, wide/defocused stimulation,focused (e.g., multipolar current focusing) stimulation, etc. Inoperation, the channel focusing control module 350 may provide focusingcontrol inputs, represented in FIG. 3 by arrow 351, to the stimulationpulse determination module 340 to set the stimulus resolution of thedelivered electrical stimulation signals based on the periodicprobability.

FIGS. 7A-7E are a series of schematic diagrams illustrating exemplaryelectrode currents and stimulation patterns for five (5) differentstimulus resolutions (i.e., different defocusing indices). It is to beappreciated that the stimulation patterns shown in FIGS. 7A-7E aregenerally illustrative and that, in practice, the stimulation currentmay spread differently in different recipients.

Each of the FIGS. 7A-7E illustrates a plurality of electrodes shown aselectrodes 728(1)-728(9), which are spaced along the recipient's cochleafrequency axis (i.e., along the basilar membrane). FIGS. 7A-7E alsoinclude solid lines of varying lengths that extend from variouselectrodes to generally illustrate the intra-cochlear stimulationcurrent 780(A)-780(E) delivered in accordance with a particular channelconfiguration. However, it is to be appreciated that stimulation isdelivered to a recipient using charge-balanced waveforms, such asbiphasic current pulses and that the length of the solid lines extendingfrom the electrodes in each of FIGS. 7A-7E illustrates the relative“weights” that are applied to both phases of the charge-balancedwaveform at the corresponding electrode in accordance with differentchannel configurations. As described further below, the differentstimulation currents 780(A)-780(E) (i.e., different channel weightings)results in different stimulation patterns 782(A)-782(E), respectively,of voltage and neural excitation along the frequency axis of the cochlea

Referring first to FIG. 7C, shown is the use of a monopolar channelconfiguration where all of the intra-cochlear stimulation current 780(C)is delivered with the same polarity via a single electrode 728(5). Inthis embodiment, the stimulation current 780(C) is sunk by anextra-cochlear return contact which, for ease of illustration, has beenomitted from FIG. 7C. The intra-cochlear stimulation current 780(C)generates a stimulation pattern 782(C) which, as shown, spreads acrossneighboring electrodes 728(3), 728(4), 728(6), and 728(7). Thestimulation pattern 782(C) represents the spatial attributes (spatialresolution) of the monopolar channel configuration.

FIGS. 7A and 7B illustrate wide or defocused channel configurationswhere the stimulation current is split amongst an increasing number ofintracochlear electrodes and, accordingly, the width of the stimulationpatterns increases and thus provide increasingly lower spatialresolutions. In these embodiments, the stimulation current 780(A) and780(B) is again sunk by an extra-cochlear return contact which, for easeof illustration, has been omitted from FIGS. 7A and 7B.

More specifically, in FIG. 7B the stimulation current 780(B) isdelivered via three electrodes, namely electrodes 728(4), 728(5), and728(6). The intra-cochlear stimulation current 780(B) generates astimulation pattern 782(B) which, as shown, spreads across electrodes728(2)-728(8). In FIG. 7A, the stimulation current 780(A) is deliveredvia five electrodes, namely electrodes 728(3)-728(7). The intra-cochlearstimulation current 780(A) generates a stimulation pattern 782(A) which,as shown, spreads across electrodes 728(1)-728(9). In general, the widerthe stimulation pattern, the lower the spatial resolution of thestimulation signals.

FIGS. 7D and 7E illustrate focused channel configurations whereintracochlear compensation currents are added to decrease the spread ofcurrent along the frequency axis of the cochlea. The compensationcurrents are delivered with a polarity that is opposite to that of aprimary/main current. In general the more compensation current at nearbyelectrodes, the more focused the resulting stimulation pattern (i.e.,the lower the width of the stimulus patterns increase and thusincreasingly higher spatial resolutions). That is, the spatialresolution is increased by introducing increasing large compensationcurrents on electrodes surrounding the central electrode with thepositive current.

More specifically, in FIG. 7D positive stimulation current 780(D) isdelivered via electrode 728(5) and stimulation current 780(D) ofopposite polarity is delivered via the neighboring electrodes, namelyelectrodes 728(3), 728(4), 728(6), and 728(7). The intra-cochlearstimulation current 780(D) generates a stimulation pattern 782(D) which,as shown, only spreads across electrodes 728(4)-728(6). In FIG. 7E,positive stimulation current 780(E) is delivered via electrode 728(5),while stimulation current 780(E) of opposite polarity is delivered viathe neighboring electrodes, namely electrodes 728(3), 728(4), 728(6),and 728(7). The intra-cochlear stimulation current 780(E) generates astimulation pattern 782(E) which, as shown, is generally localized tothe spatial area adjacent electrode 728(5).

The difference in the stimulation patterns 782(D) and 782(E) in FIGS. 7Dand 7E, respectively, is due to the magnitudes (i.e., weighting) ofopposite polarity current delivered via the neighboring electrodes728(3), 728(4), 728(6), and 728(7). In particular, FIG. 7D illustrates apartially focused configuration where the compensation currents do notfully cancel out the main current on the central electrode and theremaining current goes to a far-field extracochlear electrode (notshown). FIG. 7E is a fully focused configuration where the compensationcurrents fully cancel out the main current on the central electrode728(5) (i.e., no far-field extracochlear electrode is needed).

As noted, FIGS. 7A-7E collectively illustrate techniques for adjustingthe spatial resolution (i.e., adjusting the spatial attributes of theelectrical stimulation) based on estimated periodic probabilities, inaccordance with embodiments presented herein. However, also as noted, itis to be appreciated that other methods for altering the stimulusresolution could be used in combination with, or as an alternative to,adjustments to the spatial resolution enabled by different stimulationstrategies. For example, another technique for adapting the stimulusresolution includes varying the temporal resolution via pulse rate(i.e., higher pulse rates for higher temporal resolutions and lowerpulse rates for lower temporal resolutions) based on estimated periodicprobabilities.

Another technique for adapting the stimulus resolution based onestimated periodic probabilities includes varying the number ofstimulation sites of the stimulation by changing the number of maxima inthe channel selection. For example, the number of stimulation sites canbe increased by increasing the number of channels selected by thechannel selection module 338 and decreased by decreasing the number ofchannels selected by the channel selection module 338.

A still other technique for adapting the stimulus resolution based onestimated periodic probabilities includes varying the frequencyresolution. The frequency resolution of the filterbank module 334 can beincreased by, for example, in an FFT filterbank using a higher-pointFFT. The frequency resolution of the filterbank module 334 can bedecreased by, for example in an FFT filterbank using a lower-point FFT.

Again, returning to FIG. 4, at 467, in certain embodiments presentedherein, user-based or automatic control inputs may be used to controlthe identified sound sources, channel modulations, or focusing (e.g.,control one or more of the fundamental frequency module 342, channelmodulation module 348, or the channel focusing control module 350). Forexample, the individual sound sources that are identified and tracked bythe fundamental frequency module 342 may be controlled in a number ofways. Bilateral controls include the option to purposefully deliverdifferent pitches or sound sources to the same or different ears. Thisis done either manually by the user or may be automated based on, forexample, estimated spatial location in a real-world situation, arecorded stereo sound file, etc. Users may select sound sourcespreferentially to either enhance the pitch or loudness of a given sourceor to de-emphasize or mute the source. Sound source selection may alsobe automated using, for example, head position, gaze direction, or EEGresponses to predict the desired source of the user.

In summary, FIGS. 3 and 4 illustrate techniques that improve arecipient's perception of simultaneously received multiple frequencies(e.g., multiple F0s) included with sound signals. In particular, thetechniques presented herein identify and track multiple sound sources,extract one or more frequencies of each sound source, and, in certainembodiments enable multiple frequencies to be used to amplitudemodulation stimulation pulses delivered via different stimulationchannels. FIGS. 8, 9, and 10 illustrate further details regarding pitchencoding via amplitude modulations. For ease of illustration, FIGS. 8,9, and 10 are described with reference to the arrangement of FIG. 3.

More specifically, referring first to FIG. 8, shown is an electrodogramillustrating sequences of fixed rate stimulation pulses delivered acrossthree stimulation channels, identified as electrode 20 (E20), electrode21 (E21), and electrode 22 (E22). In FIG. 8, the stimulation pulses areeach represented by the vertical lines and are determined using onlymodules 334, 336, 338, and 340 of FIG. 3. In these examples, thestimulation pulse determination module 340 amplitude modulates thestimulation pulse sequences based only on the channel envelopes that areoutput by the envelope extraction module 336. Each of the threestimulation channels E20, E21, and E22 are modulated in substantiallythe same manner. Moreover, these pulses in FIG. 8 are generated usingmonopolar stimulation because the signal is not passed through thechannel focusing control module 350.

Referring next to FIG. 9, shown is an electrodogram illustratingsequences of fixed rate stimulation pulses delivered across threestimulation channels, identified as electrode 20 (E20), electrode 21(E21), and electrode 22 (E22). In FIG. 9, the stimulation pulses areeach represented by the vertical lines and, in these examples, the soundsignals are passed through the fundamental frequency module 342, theperiodic probability estimator module 346, and channel modulation module348. In stimulation pulse determination module 340, the pulse sequencesare amplitude modulated by the modified channel envelopes that are theoutputs of the channel modulation module 348. Additionally, in thisexample only one source, with a fundamental frequency of F0_(A), is usedto modulate the channel envelopes and, as such, each of the threestimulation channels E20, E21, and E22 are modulated in substantiallythe same manner (i.e., using fundamental frequency F0_(A)). Thestimulation pulses in FIG. 9 may be generated using varying degrees offocusing between monopolar stimulation and focused stimulation becausethe signal is passed through the channel focusing control module 350.

Referring next to FIG. 10, shown is an electrodogram illustratingsequences of fixed rate pulses delivered across three stimulationchannels, identified as electrode 20 (E20), electrode 21 (E21), andelectrode 22 (E22). In FIG. 10, the stimulation pulses are eachrepresented by the vertical lines and, in these examples, the soundsignals are passed through the fundamental frequency module 342, theperiodic probability estimator module 346, and channel modulation module348. In stimulation pulse determination module 340, the pulse sequencesare amplitude modulated by the modified channel envelopes that are theoutputs of the channel modulation module 348. In this example, threedifferent sources are used to modulate the channel envelopes. Morespecifically, the first source has a fundamental frequency of F0_(A),the second source has a fundamental frequency of F0_(B), and the thirdsource has a fundamental frequency of F0_(G). As such, as shown in FIG.10, the stimulation pulses delivered at E22 are modulated using F0_(A),the stimulation pulses delivered at E21 are modulated using F0_(B), andthe stimulation pulses delivered at E20 are modulated using F0_(C). Thestimulation pulses in FIG. 10 may be generated using varying degrees offocusing between monopolar stimulation and focused stimulation becausethe signal is passed through the channel focusing control module 350.

FIGS. 11 and 12 illustrate example scenarios with the sound environmentsinclude multiple sound sources and, as such, in which the techniquespresented herein may be implemented to improve the perception ofmultiple frequencies, such as multiple F0s. In FIG. 11, a hearingprosthesis recipient 1185 is listening to music signals that arecomprises of (include) four different sound sources, namely: recorder1186(A), guitar 1186(B), vocalist 1186(C), and drum 1186(D). In FIG. 12,the recipient 1185 is having a conversation with Speaker/Talker 1287(A)and Talker 1287(B). Meanwhile, Talkers 1287(C) and 1287(D) are holding aseparate conversation nearby. These scenarios will be used in severalexamples below.

As noted, in each of the examples of FIGS. 11 and 12, there are severalsound sources that simultaneously deliver sounds to the hearingprosthesis of recipient 1185 (e.g., multiple instruments being playedsimultaneously, multiple speakers, etc.) and, as described above, it ispossible to extract the melody, beats, and fundamental frequency ofindividual components of the sound sources. For example, in certainembodiments, measurements relating to spectral attributes such asharmonicity, spectral smoothness, etc. may be used to extract thefundamental frequencies of the different sources. In other embodiments,non-negative matrix factorization (NMF) based methods, sparse codingbased methods may be used to separate the sources and, potentially,extract the fundamental frequencies of the separated sources.

Also as described above, the techniques presented herein are able toencode the multiple frequencies in the stimulation signals delivered tothe recipient 1185. For example, when a strong sense of pitch, or highharmonic probability, is detected for a sound, the techniques presentedherein deliver deep modulations over the carrier pulses at the rate ofone of the fundamental frequencies. The techniques presented herein arenot restricted to a single F0 encoded across all channels, but insteadmultiple F0s, harmonics, etc. can be delivered across differentchannels, or groups of channels, as deep modulations.

Using the scenario in FIG. 11, the F0 from each source 1186(A)-1186(D)is tracked and extracted using the methods described above. Each F0 thatis extracted from its source will then be used to modulate the signal(s)being delivered to one or more stimulation channels. Multiple concurrentF0s can be distributed across channels according to their relativestrength in each frequency channel over time, with only a single F0assigned to any given channel at any one time (e.g., based on periodicprobability). In certain embodiments, the dominant source in a givenfrequency channel is used to select the F0 for that channel. In otherembodiments, the available channels are split amongst the detectedsources with a preference for coding each source in the channels basedon the source with the most energy. In still other embodiments, whenonly one source is being encoded, a single F0 and one or more of itsharmonics from the one musical instrument will be extracted and used tomodulate one or more channels.

Using the scenario in FIG. 12, the F0 for each source, 1287(A) through1287(D), will be tracked and extracted using the methods describedabove. Each F0 will then be used to modulate the signal being deliveredto one channel or more channels. As before, multiple concurrent F0s areto be distributed across channels according to, for example, theirrelative strength in each frequency channel over time, with a single F0assigned to any given channel at any one time.

Also as described above, to maximize pitch specificity with thetechniques presented, the above encoding strategies may be used incombination with stimulation resolution adaption techniques, such asfocused stimulation techniques. In contrast to monopolar stimulation,focused stimulation improves discrimination of spectral features andincreases the temporal independence of different frequency channels. Thecombined effects of focused stimulation and the enhanced sound codingstrategies described herein improve the ability of hearing prosthesisrecipients to discriminate and follow multiple pitches over time.

Also as described, a feature of the techniques presented herein is thatthe different sources and/or frequencies provided can be controlled in avariety of ways. For example, in certain embodiments, bilateral controlmay be provided where different frequencues or sound sources may bedelivered to different ears. For example, in the scenario of FIG. 11,the F0 from the recorder 1186(A) may be sent to one ear, and the F0 fromthe vocalist 1186(C) may be sent to the other ear. Alternatively,different harmonics from only one instrument may be separated and sentto different ears to reduce channel interactions. In another exampledescribed with reference to the scenario of FIG. 12, the F0 from Talker1287(A) may be sent only to the left ear, and the F0 from Talker 1287(B)may be sent only to the right ear. This technique may improve perceptualsegregation of different sources or pitches. Bilateral control may bedone either manually by the user or may be automated (e.g., based onestimated spatial location in a real-world situation or in a recordedstereo sound file).

In further embodiments presented herein, source selection may beprovided where the recipient is given the option to select a specificsound source. For example, in the scenario of FIG. 12, recipient 1285may select Talker 1186(B) and the F0 therefrom can be used to providedeep modulations across all channels until the desired source is changedto a different talker. Users may change sources using a “next” button onan external component, a remote control, a personal audio device, etc.to cycle through different sources. In another example described withreference to the scenario of FIG. 11, the recipient 1185 is providedwith a graphical display of the spatial arrangement of the differentsound sources, and the user is able to select the desired source usingon the graphical display.

In still further embodiments, the techniques presented herein mayprovide the recipient 1185 with the option to modify the relative volumeof different pitches or sources, such as by changing the volume ofdifferent instruments in a musical mix. For example, in the arrangementof FIG. 11, the recipient 1185 may wish to increase the vocals 1186(C)and the drum 1186(D) relative to the other instruments. For a singlemusical instrument, the recipient 1185 may change the balance betweenthe F0 and other harmonics to modify the timbre. In another exampledescribed with reference to FIG. 12, recipient 1185 may choose toincrease the volume of Talker 1187(B), who is slightly farther away thanTalker 1187(A). Recipient 1185 may also choose to decrease the volume ofTalkers 1187(C) and 1187(D) or to mute their sources entirely. Volumemodification may be done manually using an external component, a remotecontrol, a personal audio device, graphical display, etc.

FIG. 13 is a schematic block diagram illustrating an arrangement for asound processing unit in accordance with an embodiment of the presentinvention. For ease of description, the sound processing unit of FIG. 13is referred to as sound processing unit 1312.

As shown, the sound processing unit 1312 includes one or more processors1388 and a memory 1389. The memory 1389 includes multi-frequency soundprocessing logic 1325. Sound processing unit 1312 also comprises twomicrophones 1308(A) and 1308(B), one or more auxiliary input devices1309 (e.g., audio ports, such as a Direct Audio Input (DAI), data ports,such as a Universal Serial Bus (USB) port, cable port, etc.), a wirelesstransmitter/receiver (transceiver) 1311, a radio-frequency (RF)transceiver 1321, and at least one battery 1307.

The memory 1389 may be read only memory (ROM), random access memory(RAM), or another type of physical/tangible memory storage device. Thus,in general, the memory 1389 may comprise one or more tangible(non-transitory) computer readable storage media (e.g., a memory device)encoded with software comprising computer executable instructions andwhen the software, multi-frequency sound processing logic 1325, isexecuted by the one or more processors 1388, it is operable to performthe operations described herein with reference to a multi-frequencysound processor, as described elsewhere herein.

FIG. 13 illustrates software implementations for a multi-frequency soundprocessor. However, it is to be appreciated that one or more operationsassociated with the multi-frequency sound processor may be partially orfully implemented with digital logic gates in one or moreapplication-specific integrated circuits (ASICs).

FIG. 14 is a flowchart illustrating a method 1400 in accordance withembodiments presented herein. Method 1400 begins at 1402 where a hearingprosthesis receives sound signals. At 1404, a plurality of frequenciesare extracted from the sound signals and, at 1406, the sound signals arefiltered to generate channelized sound signals. At 1408, a plurality ofstimulation pulse sequences are determined, where each stimulation pulsesequences corresponds to one of the channelized sound signals. At 1410,each of the plurality of stimulation pulse sequences are amplitudemodulated with one of the plurality of frequencies extracted from thesound signals. At least two of the plurality of stimulation pulsesequences are amplitude modulated with different ones of the pluralityof frequencies extracted from the sound signals. At 1412, each of theplurality of stimulation pulse sequences are delivered to the recipientvia one or more stimulation channels of the hearing prosthesis.

FIG. 15 is a flowchart illustrating a method 1500 in accordance withembodiments presented herein. Method 1500 begins at 1502 where a hearingprosthesis receives sound signals. At 1504, at least one frequency isextracted from the sound signals and, at 1506, the sound signals arefiltered to generate channelized sound signals. At 1508, a plurality ofstimulation pulse sequences are determined, where each of the pluralityof stimulation pulse sequences corresponds to one of the channelizedsound signals. At 1510, a periodic probability is determined for each ofa plurality of the channelized sound signals, where a periodicprobability indicates a degree of association between a channelizedsound signal and the at least one frequency extracted from the soundsignals. At 1512, at least one of the plurality of stimulation pulsesequences is amplitude modulated based on a periodic probabilityassociating a channelized sound signal corresponding to the at least onestimulation pulse sequence and the at least one frequency extracted fromthe sound signals.

It is to be appreciated that the above described embodiments are notmutually exclusive and that the various embodiments can be combined invarious manners and arrangements.

The invention described and claimed herein is not to be limited in scopeby the specific preferred embodiments herein disclosed, since theseembodiments are intended as illustrations, and not limitations, ofseveral aspects of the invention. Any equivalent embodiments areintended to be within the scope of this invention. Indeed, variousmodifications of the invention in addition to those shown and describedherein will become apparent to those skilled in the art from theforegoing description. Such modifications are also intended to fallwithin the scope of the appended claims.

What is claimed is:
 1. A method, comprising: receiving sound signals ata hearing prosthesis; identifying a plurality of sound sourcesassociated with the sound signals; extracting a plurality of fundamentalfrequencies from the sound signals, wherein each of the plurality offundamental frequencies is associated with a different one of theplurality of sound sources; tracking, over time, sound componentsassociated with each of the plurality of sound sources based on theplurality of fundamental frequencies; filtering the sound signals togenerate channelized sound signals; determining a plurality ofstimulation pulse sequences, wherein each of the plurality ofstimulation pulse sequences corresponds to one of the channelized soundsignals; amplitude modulating a first one of the plurality ofstimulation pulse sequences with a first fundamental frequencyassociated with a first one of the plurality of sound sources; amplitudemodulating a second one of the plurality of stimulation pulse sequenceswith a second fundamental frequency associated with a second one of theplurality of sound sources; and delivering the plurality of stimulationpulse sequences to a recipient of the hearing prosthesis via one or morestimulation channels of the hearing prosthesis.
 2. The method of claim1, further comprising: determining a periodic probability for each of aplurality of the channelized sound signals, wherein a periodicprobability indicates a degree of association between a channelizedsound signal and a fundamental frequency extracted from the soundsignals; and setting a stimulus resolution of at least one of theplurality of stimulation pulse sequences based on a periodic probabilityassociating a channelized sound signal corresponding to the at least oneof the plurality of stimulation pulse sequences and at least onefundamental frequency extracted from the sound signals.
 3. The method ofclaim 2, wherein setting a stimulus resolution of the at least one ofthe of the plurality of stimulation pulse sequences comprises: setting aspatial resolution of the at least one of the of the plurality ofstimulation pulse sequences based on the periodic probability determinedfor the corresponding channelized sound signal.
 4. The method of claim1, further comprising: determining, over time, a number of stimulationpulse sequences for tracked sound components associated with the firstone of the plurality of sound sources; and delivering all of the numberof stimulation pulse sequences for the tracked sound componentsassociated with the first one of the plurality of sound sources to therecipient via a same stimulation channel.
 5. The method of claim 1,wherein the hearing prosthesis comprises a first implantable componentconfigured to be implanted at a first ear of the recipient and a secondimplantable component configured to be implanted at a second ear of therecipient, and wherein the method further comprises: determining, overtime, a number of stimulation pulse sequences for tracked soundcomponents associated with the first one of the plurality of soundsources; determining, over time, a number of stimulation pulse sequencesfor tracked sound components associated with the second one of theplurality of sound sources; delivering all of the number of stimulationpulse sequences for the tracked sound components associated with thefirst one of the plurality of sound sources to the recipient via thefirst component; and delivering all of the number of stimulation pulsesequences for the tracked sound components associated with the secondone of the plurality of sound sources to the recipient via the secondcomponent.
 6. The method of claim 1, further comprising: extracting atleast one harmonic of the fundamental frequency of at least one soundsource of the plurality of sound sources, amplitude modulating at leastone of the plurality of stimulation pulse sequences with the at leastone harmonic of the fundamental frequency of the at least one soundsource.
 7. The method of claim 1, further comprising: determining aperiodic probability for each of a plurality of the channelized soundsignals, wherein a periodic probability indicates a degree ofassociation between a channelized sound signal and a fundamentalfrequency extracted from the sound signals, wherein an amplitudemodulation applied to at least one stimulation pulse sequence is basedon a periodic probability associating a channelized sound signalcorresponding to the at least one stimulation pulse sequence and atleast one fundamental frequency extracted from the sound signals.
 8. Themethod of claim 7, wherein determining a periodic probability for eachof the plurality of the channelized sound signals, comprises: for afirst one of the channelized sound signals, determining a plurality ofperiodic probabilities that each associate one of the plurality of thefundamental frequencies with the first channelized sound signal, whereinan amplitude modulation applied to a first stimulation pulse sequence isbased on a determination of the plurality periodic probabilitiesassociating the plurality of the fundamental frequencies with the firstchannelized sound signal.
 9. The method of claim 1, further comprising:receiving an input from a user; and determining an amplitude modulationapplied to at least one of the plurality of stimulation pulse sequencesis based on the input from the user.
 10. A method, comprising: receivingsound signals at a hearing prosthesis; identifying a plurality of soundsources associated with the sound signals; extracting a plurality offundamental frequencies from the sound signals, wherein each of theplurality of fundamental frequencies are associated with a different oneof the plurality of sound sources; tracking, over time, sound componentsassociated with each of the plurality of sound sources based on theplurality of fundamental frequencies; filtering the sound signals togenerate channelized sound signals; determining a plurality ofstimulation pulse sequences, wherein each of the plurality ofstimulation pulse sequences corresponds to one of the channelized soundsignals; determining a periodic probability for each of a plurality ofthe channelized sound signals, wherein the periodic probabilityindicates a degree of association between a channelized sound signal andat least one fundamental frequency of the plurality of fundamentalfrequencies extracted from the sound signals; amplitude modulating afirst one of the plurality of stimulation pulse sequences with a firstfundamental frequency associated with a first one of the plurality ofsound sources based on the periodic probability associating achannelized sound signal corresponding to the first stimulation pulsesequence and the first fundamental frequency; and amplitude modulating asecond one of the plurality of stimulation pulse sequences with a secondfundamental frequency associated with a second one of the plurality ofsound sources based on the periodic probability associating achannelized sound signal corresponding to the second stimulation pulsesequence and the second fundamental frequency.
 11. The method of claim10, further comprising: setting a stimulus resolution of the at leastone of the plurality of stimulation pulse sequences based on theperiodic probability associating the channelized sound signalcorresponding to the at least one stimulation pulse sequence and the atleast one fundamental frequency extracted from the sound signals. 12.The method of claim 11, wherein setting a stimulus resolution of the atleast one of the of the plurality of stimulation pulse sequencescomprises: setting a spatial resolution of the at least one of the ofthe plurality of stimulation pulse sequences based on the periodicprobability determined for the corresponding channelized sound signal.13. The method of claim 11, further comprising: extracting at least oneharmonic of the fundamental frequency of at least one sound source ofthe plurality of sound sources, amplitude modulating at least one of theplurality of stimulation pulse sequences with the at least one harmonicof the fundamental frequency of the at least one sound source.
 14. Ahearing prosthesis, comprising: one or more sound input elementsconfigured to receive sound signals; a memory; a stimulator unit; and atleast one processor configured to: identify at least first and seconddifferent sound sources associated with the sound signals, estimate atleast first and second different fundamental frequencies present withinthe received sound signals, the first and second different fundamentalfrequencies being respectively associated with the first and seconddifferent sound sources, track, over time, at least first and secondsound components, which are respectively associated with the first andsecond different sound sources, the first and second sound componentsbeing tracked based on the estimated first and second differentfundamental frequencies, determine at least first and second stimulationpulse sequences based on the sound signals, amplitude modulate the firststimulation pulse sequence with the first fundamental frequencyassociated with the first sound source, and amplitude modulate thesecond stimulation pulse sequence with the second fundamental frequencyassociated with the second sound source, wherein the stimulator unit isconfigured to deliver the first and second stimulation pulse sequencesto a recipient of the hearing prosthesis.
 15. The hearing prosthesis ofclaim 14, wherein the first and second stimulation pulse sequencescorrespond to first and second channelized sound signals, respectively,determined from the sound signals, and wherein the at least oneprocessor is configured to: determine a periodic probability for atleast the first channelized sound signal, wherein the periodicprobability indicates a degree of association between the firstchannelized sound signal and the first estimated fundamental frequencyof the sound signals.
 16. The hearing prosthesis of claim 15, whereinthe at least one processor is configured to: set a stimulus resolutionof at least the first stimulation pulse sequence based on the periodicprobability associating the first channelized sound signal with at leastthe first fundamental frequency estimated from the sound signals. 17.The hearing prosthesis of claim 16, wherein to set a stimulus resolutionof the first stimulation pulse sequence, the at least one processor isconfigured to: set a spatial resolution of the first stimulation pulsesequence based on the periodic probability associating the firstchannelized sound signal with at least the first fundamental frequencyestimated from the sound signals.
 18. The hearing prosthesis of claim15, wherein the at least one processor is configured to: set anamplitude modulation applied to at least the first stimulation pulsesequence based on the periodic probability associating the firstchannelized sound signal with at least the first fundamental frequencyestimated from the sound signals.